Skip to main content

Data Loaders

Data Iterator​

class eole.inputters.dynamic_iterator.DynamicDatasetIter(corpora, corpora_info, transforms, vocabs, task, batch_type, batch_size, batch_size_multiple, data_type='text', bucket_size=2048, bucket_size_init=-1, bucket_size_increment=0, device=device(type='cpu'), skip_empty_level='warning', stride=1, offset=0, score_threshold=0, left_pad=False, model_type=None, image_patch_size=16, image_size=1024, adapter='llava', pooling_kernel_size=1, mm_tokens_per_image=280, sample_rate=16000, num_mel_bins=80, n_fft=400, hop_length=160, chunk_length=30)​

Bases: IterableDataset

Yield batch from (multiple) plain text corpus.

  • Parameters:
    • corpora (dict *[*str , ParallelCorpus ]) – collections of corpora to iterate;
    • corpora_info (dict *[*str , dict ]) – corpora infos correspond to corpora;
    • transforms (dict *[*str , Transform ]) – transforms may be used by corpora;
    • vocabs (dict *[*str , Vocab ]) – vocab dict for convert corpora into Tensor;
    • task (str) – CorpusTask.TRAIN/VALID/INFER;
    • batch_type (str) – batching type to count on, choices=[tokens, sents];
    • batch_size (int) – numbers of examples in a batch;
    • batch_size_multiple (int) – make batch size multiply of this;
    • data_type (str) – input data type, currently only text;
    • bucket_size (int) – accum this number of examples in a dynamic dataset;
    • bucket_size_init (int) – initialize the bucket with this
    • examples; (size with this amount of)
    • bucket_size_increment (int) – increment the bucket
    • examples;
    • skip_empty_level (str) – security level when encouter empty line;
    • stride (int) – iterate data files with this stride;
    • offset (int) – iterate data files with this offset.
  • Variables:
    • sort_key (function) – functions define how to sort examples;
    • mixer (MixingStrategy) – the strategy to iterate corpora.

batch_iter(data, batch_size, batch_type='sents', batch_size_multiple=1)​

Yield batches from data, with each batch size a multiple of batch_size_multiple.

  • Parameters:
    • data – Iterable of (example, index) tuples
    • batch_size – Target batch size
    • batch_type – “sents” or “tokens”
    • batch_size_multiple – Batch size must be multiple of this
  • Yields: List of (example, index) tuples forming a batch

classmethod from_config(corpora, transforms, vocabs, config, task, device, stride=1, offset=0, model_type=None)​

Initilize DynamicDatasetIter with options parsed from opt.

class eole.inputters.dynamic_iterator.MixingStrategy(iterables, weights)​

Bases: object

Mixing strategy that should be used in Data Iterator.

class eole.inputters.dynamic_iterator.SequentialMixer(iterables, weights)​

Bases: MixingStrategy

Generate data sequentially from iterables which is exhaustible.

class eole.inputters.dynamic_iterator.WeightedMixer(iterables, weights, worker_id=None)​

Bases: MixingStrategy

A mixing strategy that mix data weightedly and iterate infinitely.

Dataset​

class eole.inputters.text_corpus.ParallelCorpus(name, path_src, path_tgt, path_sco=None, path_align=None)​

Bases: object

A parallel corpus file pair that can be loaded to iterate.

load(offset=0, stride=1)​

Load file and iterate by lines. offset and stride allow to iterate only on every stride example, starting from offset. In the case of local files, all files are open exactly the same way by each worker Therefore we need to apply a stride / offset rule to make sure we do not process the same ex. In the case of HF streaming mode we need to make sure we have more shards than workers. Typically we recommend to have shard being a multiple of workers for instance for big datasets: 16 shards for 4 workers. The shards will be iterated automatically since HF locks shards when in use.

class eole.inputters.text_corpus.ParallelCorpusIterator(corpus, transform, skip_empty_level='warning', stride=1, offset=0, is_train=False)​

Bases: object

An iterator dedicated to ParallelCorpus.

  • Parameters:
    • corpus (ParallelCorpus) – corpus to iterate;
    • transform (TransformPipe) – transforms to be applied to corpus;
    • skip_empty_level (str) – security level when encouter empty line;
    • stride (int) – iterate corpus with this line stride;
    • offset (int) – iterate corpus with this line offset.